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1. INTRODUCTION 

Infrastructure is required to develop and operate an application with micro-service architecture. As 
an application, the micro-services are expected to work together to provide a larger function of services in 
which the connectivity between the services becomes critical. To ensure the communication of each service, 
a reliable network is needed. Security acts to protect the services by validating and mitigating these services 
from being compromised or attacked. Virtualization is one of the infrastructure technologies that is 
commonly used by enterprises to host applications, including services. Alternatively, the enterprises could be 
benefited by applying container technology resulting in immutable and agile services. Explain software aging 
in cloud computing, software aging is in the operation software systems will be accumulate errors to system 
failure and other hazardous consequences [1]. Software aging can be happening because memory 
fragmentation, resource consumption in large scale and accumulation of errors happen in system. 

One of the Indonesian financial service enterprises has deployed application container technology as 
infrastructure technology. An IT security head explains that his enterprise as well as many other enterprises are 
striving to be more competitive in their field. Reliable technology infrastructure enables his enterprise to 
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provide significant improvement, such as shortening the product-development cycles. Moreover, 
troubleshooting and manual setting of the configuration changes would require less time. The respondent also 
focuses on important aspects of infrastructure, such as network and security. His concern raises two major 
questions: 1) “how to run applications on Kubernetes container technology?” and ii) “how to address the 
network and security aspects in their highly virtualized data center?”. Using hypervisors, most of the 
enterprises’ applications are run virtually while the dynamic routing protocol is used to route the access to the 
applications in a physical network device by the open shortest path first (OSPF). A firewall to inspect outgoing 
and incoming traffic is deployed to implement security at the perimeter. In this case study, Kubernetes acted as 
leverage only in the farm segment of the enterprise’s internal server, instead of the entire network segment. 

According to Pahl [2], the container could be activated via a requirement to deploy an application on 
distributed cloud platforms because many enterprises have adopted Kubernetes to run their applications. 
Kubernetes along with Docker is a standard platform for a cloud-based service [3]. Moreover, Pahl discusses 
the relevance of containers with virtualization technology. The relevance is laid on the requirement of 
containers for advanced networking features based on network virtualization like routing [2]. To manage 
applications on a microservice basis, Kubernetes allows the creation of multiple Horizontal Pod Autoscaler 
instances adapts to a single microservice deployment, and proposes me-Kube (multi-level elastic 
Kubernetes), referring to a Kubernetes extension that introduces a hierarchical architecture to control the 
elasticity of application-based microservices [4]. Containers are characterized by having the ephemeral state 
and aspects of a stateful microservice, which create a more complex array previously created by Kubernetes 
controllers [5]. The development of microservice architecture-based cloud applications presents a variety of 
challenges; one of them is scalability at the container level. A study shows that an algorithm uses an agnostic 
approach to auto-scale microservices which are implemented in Google Kubermetes Engine [6]. 
Microservices in software development arise when a business department finds risk in a product and starts 
asking questions; an example of the migration from monolithic to microservices took place in 2010 when 
Netflix started using Amazon's Web Services (AWS) to host applications and services with over 100 
fine-grained services instead of web applications, also known as or war [7]. 

Since the new threats using techniques to bypass perimeter-based protections emerge, John Becker 
highlights the significance of the Forrester® zero trust model and explains the vulnerability of IP-based 
network perimeter [8]. An American market research company, Forrester®, proposes the zero trust model 
stipulating that all networks should be classified as untrusted, and all traffic must be logged and inspected 
without considering its location; these strategies will lead to the use of least privilege strategies and enforce 
strict access control [9]. Zero trust model can be used to secure and protect the big data environment as well 
as solve a problem in big data security [10]. A zero trust management model for internet of things (IoT) can 
guarantee the validation and participation of the credentials and configurations of every resource of the 
infrastructure in the network [11]. A study proposes a model that is applicable to protect various data adopted 
from the zero trust model and present new techniques based on the zero trust model to secure the data [12]. 

The implementation of zero trust for the energy sector could reduce the risks, secure the systems, 
and protect the data of zero trust security architecture [13]. The authors have developed cyber security and 
combined the zero trust model with cloud computing because zero trust schemes in cloud computing should 
be integrated and combined with practices, policies, and technologies; thus, cyber security could become 
workable [14]. The combination of the zero trust model and Kubernetes can protect infrastructure from 
attackers who threaten the data of enterprises or organizations [15]. 

Plenty of security aspects and container systems are still necessarily explored. This study provides a 
novel network design by developing secure Kubernetes based on container technology. This development is 
expected to serve several recommendations for the enterprise. The vendors’ best practices, such as the 
network virtualization technology of VMware and Cisco, were reviewed and adopted as a part of our design. 
The zero trust model of Forrester® was also outlined as the security guideline. To validate the proposed 
design, a lab simulation was deployed to explore the ability of the design to fulfill the enterprises’ 
requirements. This paper is an extended version of the work of [16]. 


2. PROBLEM STATEMENT AND PRELIMINARIES 
2.1. Kubernetes container system 

Kubernetes is used over the years to suffice the needs and the capability efficiently and using 
Kubernetes to expose HTTP endpoints for metrics as it is simpler and most used [17]. Proposed work to 
move high computational power near IoT sensors with fog serverless framework (FSF) and using Kubernetes 
to form an interface with Kubernetes cluster which manage a set of Docker container [18]. Each of the 
Kubernetes clusters has one or more worker nodes and masters [3] because the namespaces of each cluster 
enable the isolation of networks and environments. Meanwhile, different environments, such as development 
and production, can be segmented to have their network segment in traditional physical networks. In 
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Kubernetes, similar levels of segmentation possibly use a namespace. Kubernetes et al. [3] define a pod as 
the smallest entity in the Kubernetes cluster. The container pod is allocated with its internet protocol (IP) 
address, which is comparable to the endpoints, such as a real machine in the physical network environment. 
A network gateway or router is required for the container pods in a namespace. Thus, the external network is 
accessible to external networks or other namespaces. The containers run within pods in Kubernetes. 
Nonetheless, Kubernetes does not manage the containers within a pod because it only manages the pods. 
Through the master node, Kubernetes could expand or shrink the number of pods [3]. The pod’s probability 
of compromising would increase as the number of pods increases. The zero trust model is implemented at the 
network level to mitigate the risk of compromising, internal attacks, and external attacks as well as ultimately 
avoid disruption to the business environment [9]. 


2.2. Zero trust model-based information security 

There are no trusted and untrusted-internal or external networks in the zero trust model [9]. 
Inspection and logging should be performed on all traffic since the traffic is categorized as untrusted. 
According to Forrester®, the perimeter-based network security insufficiently protects the networks and is 
labeled as internal or trusted to perimeter-based protection [9]. The virtual local area network (VLAN), which 
performs network segmentation, is not categorized as a network security method since it cannot prevent any 
traffic to move between VLANs and cannot gain access to critical systems in different VLANs [19]. 

The architecture that deploys microservice applications employs Kubernetes in the private cloud and 
high availability of Kubernetes for application-based microservices [20]. As a comparison, the 2017 IBM 
X-Force report has also been studied. The results of investigating global threat intelligence and deep security 
could provide enhanced security solutions. The report states that in 2016, 58% of the attacks in the financial 
service sector is from insiders while 53% of these attacks are not noticed by any employees [21]. The study 
on Forrester® and the 2017 IBM X-Force report provide a strong motivation to more comprehensively 
implement the zero trust model in the network. Meanwhile, adopts the zero trust model in intranet data to 
secure the authentication when users want to access intranet data [22]. This method or architecture can more 
effectively protect network communication. 


3. THE PROPOSED METHOD DESIGN OF NETWORK AND SECURITY 

This study employed a top-down design approach [23] to design the network. High-level staff, such 
as the Head of the Department, were targeted from the financial service enterprises in Indonesia to 
understand the business constraints and goals. Meanwhile, the manager-level staff contributed to give 
knowledge on the technical goals and limitations to produce a proposed design. Then, the lab environment 
was built to ensure that all technical requirements had been achieved. The requirements were based on the 
enterprises’ business requirements to collect the technical requirements. The container network connectivity 
from and to existing external physical networks was tested by dynamic-routing protocols to conduct route 
redistribution. By default, all of the traffics were considered unsecured or untrusted, unless they had been 
stated. In other words, the network security was configured to only allow intended traffic between containers. 

In detail, the plan, design, implement, operate, and optimize (PDIOO) network life-cycle was 
adopted [23] to design a secure network. As shown by Figure 1, the methods were classified into three major 
steps: i) collecting requirements, ii) designing a network, and iii) evaluating the design. The results of this 
analysis are as follows. 


3.1. First stage: requirement gathering 

The process of requirement gathering contains several steps, such as defining business goals, 
translating the business goals into technical constraints and goals, and ultimately evaluating the currently 
existing network. The outputs of every process are outlined. The analysis focuses on both business and 
technical aspects. XYZ Bank’s current network topology and configuration is required, to understand how 
the container networks shall be integrated into the existing networks. 


3.1.1. Business goals 

The IT security head expects shorter product development cycles because the released cycle could 
be adjusted to the business requirement. To mitigate problems, a new secure network design should be able 
to solve inconsistent policies and causes of misconfigurations. The network and security design should 
leverage the dedicated technology for the container system, instead of using hardware-based technology. To 
accommodate such a request, the containers should run with a consistent scale that is parallel to its host and 
prefers network virtualization. 
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Design Evaluation A i , R 
* Verify design meets business and technical goals 


Figure 1. Research method 


3.1.2. Technical goals 

The technical team was interviewed to technically translate the goals by seeking their expectations 
and technical perspectives. The technical goals provided by the network team are i) aligning with the existing 
IP address schemes, ii) redistributing authentic and intended routing information through a routing protocol, 
and ili) preventing reconnaissance attacks (e.g., port scanning) and network-based attacks (e.g., route 
hijacking). Hence, internal network communications could be protected. The translation from business goals 
to technical goals is outlined in Table 1. 


Table 1. The mapping of the business goals into technical goals 


Available business goals Translation of technical goals 
Shortening life-cycles of the product Planning IP address for containerized application networks 
development Ensuring the connectivity to the existing IPv4 networks 


Mitigating problems due to security and Protecting the internal container-to-container communications 

network (e.g., misconfigurations and Ensuring only intended and authentic-routing information advertised in the networks 

inconsistent policies) Implementing network security to combat network-based attacks, including 
reconnaissance-based and route hijacking attacks. 

Modernizing container technology- Protecting the internal communications between containers 

specific technologies Implementing network security to prevent network-based attacks, such as route 
hijacking and reconnaissance-based attacks 
Properly recording traffic 


3.1.3. Technical constraints 

The IT security head expects a conformable design with the existing data center environment of the 
enterprise without additional hardware. Hence, VMware’s virtualization is deployed to virtualize the 
infrastructure, while any required virtual network components can be supported by the available hardware 
resources. The network team is also put in another constraint because the OSPF must be used in its networks 
as a dynamic routing protocol to avoid any configuration changes that might impact the existing systems; for 
example, changing routing protocols or migrating network gateways. All of the prerequisites for 
configuration changes are provided in Table 2. 
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Table 2. Necessary requirements prior to configuration change 


Pre-requisite design decision Realization of design 
Minimal having three different port groups in the Traffic differentiation and routing among physical servers in 
VDS in the form of overlay, management, and data physical networks 
Reusing the existing subnets and VLAN ID of the Aligning the security policy of XYZ Bank and putting the 
management network management IP address of all components under the out-of-band 
subnet 
Provisioning a new overlay traffic network Traversing devices and links to increase the MTU size by 1580 


bytes or above ue to the additional 80 bytes of payload in the 
original packet by overlay 
Provisioning a new network for external traffic data Isolating the high-security risk of rating in- and outgoing traffic of 
path the container networks from management and overlay’s network 


3.2. Second stage: proposed design of network 

This section discusses the network design while the next section discusses the security design. In 
this study, all of the required components of the design are virtual-based to avoid any additional hardware. 
Additional configuration changes are still required to implement the design in the existing devices. 

To set up a lab simulation for designing the study, the minimum requirements defined in the 
Kubernetes documentation [3] are used since the size and number of workloads performing in a Kubernetes 
cluster are unique for each enterprise. Kubernetes pods use the application programming interface (API) server 
in master to worker nodes within the cluster because each Kubernetes cluster ran with one worker and one 
master node [3]. Configuration changes are required when installing the underlying equipment. For example, 
the addition of VLAN and IP addressing network and the increase of maximum transmit unit (MTU) size are 
needed to add the IP packet header of the overlay network since generic network virtualization encapsulation 
(GENEVE) [24] is used to overlay encapsulation. Table 2 summarizes the configuration changes for the 
design recommendations, which relate to the design of the existing logical network. 

Figure 2 depicts that the setup of the logical container network is outlined and a high-level 
visualization of the network design is provided. Through VMware’s NSX [21], each namespace in the cluster 
provides a logical switch, and a Tier-1 router is constructed. Before being connected to the external virtual 
router, the namespaces are routed and aggregated to a Tier-O router [21]. To set up the logical network as 
shown in Figure 2, several steps are followed, such as making a design decision based on the IP address 
planning as well as routing the protocol option and security considerations. Table 3 summarizes 
recommendations for the logical network design. 


ns: kube-public | 


Logical Switch i 4 Logical Switch 3 j Logical Switch 


Master 


Overlay 


Figure 2. Proposed design of logical network 
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Table 3. The recommendation for logical network design 

Logical network design decisions Realization of the design 
Pre-allocating the 16 prefixes of network 256 subnets are reserved for new namespaces. 
address spaces for the container 
environment 
Providing at least one logical switch for Ensuring pod-to-pod network connectivity [25] 
each namespace [25] 
Using BGP as the protocol for dynamic Proving dynamic routing protocols for >100,000 


routing [26] endpoints to deploy large-scale data centers [26] 

Enabling authentication of a routing Avoid unauthorized access or/and routing information 
protocol between routing peers [27] tampers and attacks, such as route hijacking 

Enabling route filtering for containers of | Not advertising container networks without access from or 
internal networks [27] to external networks outside the domain of BGP routing 


Enterprise probably has various space availability allocations of network addresses. It is 
recommended to allocate a large prefix per cluster to aid the subnet assignment for each newly created 
namespace. Using this setup, a prefix allocation of/16 becomes a sample instead of a definitive requirement. 
Moreover, 256 namespaces could be served in a Kubernetes cluster, and 254 pods could be instantiated for 
each namespace. The modification of prefix and subnet allocations could be performed when required. 

Container podscouldbe dramaticallycompared to virtual or physical machines since the container 
pods could be easily escalated. To accommodate the dynamic networking behavior, a scalable dynamic 
routing protocol is preferred. Border gateway protocol (BGP), as a proven dynamic routing protocol, is 
reported in IETF RFC 7938, supporting “large-scale” data centers with over 100,000 servers [26]. In the 
design, the container pods are assumed as endpoints, which are similar to traditional servers and have their 
unique IP address. Kubernetes could be considered “large-scale” since it supports >100,000 pods within a 
cluster. Cisco suggests that the security mechanism of BGP in an enterprise environment (e.g., filtering and 
routing protocol authentication) could diminish the risks of attacks from the router’s perspective, such as 
route hijacking [27]. To reduce risks from reconnaissance attacks, the routes (i.e., internal pods or namespace 
networking) are not necessarily and externally advertised because they can be filtered out at the Tier-0 router. 


3.3. Third stage: proposed design of network security 

The zero trust model considers that all traffic is untrusted or unsecured [9], [19]. The inspection is 
required, and the traffic is only allowed when needed, for example when the application or service requires 
the traffic to function properly [9], [19]. Moreover, all traffic could be denied by default and only allow a 
required or legitimate traffic communication. 

The network security recommendations based on the zero trust model are summarized in Table 4. To 
inspect network traffic, the firewall could be deployed [28]. The incoming traffic from or to external 
networks could be inspected and protected by the perimeter-based firewall [9]. The firewall must be able to 
recognize the incoming traffic from the pods to execute the internal firewall at the container pod level. To 
map the pods as security objects in the firewall, Kubernetes’ label is utilized as an identifier [21], and the 
label is assigned using Kubernetes API-server to the pods after or while the pods are instantiated [3], [21]. 
The firewall policies should have been configured and defined before the pod labeling process to protect and 
inspect the incoming traffic from pods [21]. The firewall policies deny all traffic by default and only allow 
the required communications. The policies include routing protocol of the traffic, legitimate workload traffic 
or users, and the management of traffic from workloads’ or users’ requirements. Then, all traffic that has 
been analyzed is reported to an external Syslog server in order to store connectivity information. 


Table 4. The recommendation for security design 
Decision on security design Realization of the design 
Using zero trust as a model Treating all traffic as untrusted or unsecured without considering its direction or location but 
complying with the requirement of the respondents to protect external elements (at the 
perimeter) and internal elements (i.e., between container pods) [9], [19] 
Inspecting all traffics [19] Attacking unauthorized personnel mitigation in accordance with the zero trust model [19] 
Logging all traffic [19] Troubleshooting, analyzing, and/or forensic testing based on the zero trust model [19] 


4. RESULTS AND DISCUSSION 

The conformability of the proposed design with the technical requirements has been evaluated. 
Table 5 shows that six requirements have been evaluated from the security or network perspectives. 
Therequirement analysis is done to state the technical requirements as shown in Table 1. A lab simulation 
was performed using a computer with a core 4 processor, x86, 500 GB disk capacity, 32 GB RAM, and a 
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network interface card (NIC) for network connectivity. Figure 3 shows the design of the logical network for 
the lab simulation. 


A vSphere ESXi is installed to run all the required components on the computer. As a virtual 


machine in a VMware vSphere environment, a Kubernetes cluster with 2 worker nodes and 1 master node [3] 
is installed by following the setup installation guide. To set up the NSX environment and components, 
VMware’s installation guide is also followed along with Kubernetes nodes. A virtual router (CSR1Kv) is 
incorporated into the environment to simulate the layers of three devices, which are the enterprise’s 
checkpoint firewall appliances. To evaluate the design, the configurations following the proposed and 
recommended design are made. The design was evaluated using the performed connectivity tests and the 
design verifications. The results and the technical requirements are finally mapped, as presented in Table 1. 
The design evaluation is summarized in Table 5. 


Table 5. Summary of the design evaluation 


Design 


Status (successful 


Technical specification Method 3 Description 

category or failed) 

Network Planning IP address to Producing several new Successful Large network address space should be 
apply container networks namespaces in synchronized with the number of 

Kubernetes clusters namespaces, and every new namespace 
[21] is supplied with a /24 network. 

Network Securing connectivity to Redistributing BGP Successful BGP to OSPF redistributions are 

the existing IPv4 networks routes into OSPF needed at the existing layers of three 
networks devices that become checkpoints of 
firewall appliances because the 

container networks use BGP. 

Network Ensuring only intended and Conducting BGP Successful The BGP authentications should be 
authentic routing authentication and configurated at the checkpoint firewall 
information advertised in filtering out internal appliances. The internal route of 
networks routes from being namespaces is only added manually 

externally advertised into the lists of prefixes by denying the 

[27] policy at the container’s perimeter (i.e., 
Tier-0) router and restricting it from 
being broadcasted to the external 
networks. 

Security Preventing penetration Nmap port scanning of Successful A perimeter-based firewall is required 
from external routes to the the container pods to inspect outgoing and incoming traffic 
environment of the from the external from the container networks’ external 
application container network [29] routes. 
network 

Security Considering all of the Testing ICMP between Successful Allocating Kubernetes labels to every 
traffic as unsecured and container pods in the instantiated pod is compulsory to apply 
only allowing required same namespace and the firewall policy. 
container communications network [21] Moreover, proper configuration and 

documentation are needed for the 
firewall policy. 

Security Properly logging and Verifying the logging Successful External or third-party Syslog servers 


recording traffic 


of inspected traffic to a 
Syslog serve 


should store the firewall's logged traffic 
inspection logs. 


Physical Computer 


Modem + Switch 


gateway: 192.168.31.1 


Internet 


NSX Edge-to-CSRIKv 
33.33.33.32/30 
VLAN 33 


Figure 3. Design of logical network for lab environment 
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The first step of proposed design evaluations is to ensure the advertised information in the networks 
is only for the intended routing and authentic information. A new namespace is created to verify whether a 
subnet is allocated from the pre-allocated prefix and justified its external and internal connectivity. The 
external connectivity is verified using route redistribution, which refers to the virtual router of this setup and 
is performed at the external router peer. Alternatively, it can be done through the checkpoint firewalls in the 
enterprise’s environment. Moreover, BGP message digest 5 (MD5) authentication is deployed to confirm the 
authenticity of the BGP peers and filter out certain internal namespace networks. Therefore, the BGP MD5 
authentication is not acknowledged from the virtual router’s routing table. Figure 4 shows the BGP routing 
table from the virtual router before the traffic filter. 

The routing information from its peer is listed. The namespace’s subnet contains the 10.1.6.0/24 
network in the Kubernetes cluster. Figure 4 presents that the route could be filtered out by using and mapping 
the prefix lists to the outgoing interface at the BGP configuration of the Tier-0 (TO) router, supposing that the 
subnet is only internal and does not need external connectivity [27]. The 10.1.6.0/24 route would be excluded 
at the TO router after the prefix lists are mapped to the router. Figure 5 shows that the 10.1.6.0/24 could not 
be found in the virtual router’s routing table. To sum up, the external BGP authentication and filtration of the 
“internal” routes are successfully advertised. 


root@k8s-master1: ~ — ssh root@192.168.31.90 telnet 192.168.31.254 


CSR1@0@vashow ip bgp summary 

BGP router identifier 1.1.1.1, local AS number 65000 

BGP table version is 252, main routing table version 252 

9 network entries using 2232 bytes of memory 

9 path entries using 1080 bytes of memory 

3/3 BGP path/bestpath attribute entries using 744 bytes of memory 
1 BGP AS-PATH entries using 24 bytes of memory 

© BGP route-map cache entries using @ bytes of memory 

® BGP filter-list cache entries using Ô bytes of memory 

BGP using 4080 total bytes of memory 

BGP activity 37/28 prefixes, 38/29 paths, scan interval 6@ secs 


Neighbor v AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 
33.33.33.34 4 65001 79 82 252 e 0 01:04:21 6 
CSR1000v#show ip bop 

BGP table version is 252, local router ID is 1.1.1.1 

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
RI8-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
best-external, a additional-path, c RIB-compressed, 

Origin codes: i - IGP, e - EGP, ? - incomplete 

RPKI validation codes: V valid, I invalid, N Not found 


xn 


Network Next Hop Metric LocPrf Weight Path 
*> 1.1.1.1/32 0.0.0.0 @ 32768 i 
*> 7.7.7.7/32 0.0.0.0 @ 32768 ? 
+> 10.0.99.0/24 33.33.33.34 0 65001 ? 
*> 10.1.3.0/24 33.33.33.34 @ 65001 ? 
+> 10.1.4.0/24 33.33.33.34 © 65001 ? 
*> 10.1.5.0/24 33.33.33.34 O 65001 ? 
*> 10.1.6.0/24 33.33.33.34 © 65001 ? 
*> 172.16.0.0/24 33.33.33.34 @ 65001 ? 
*> 192.168.31.0 0.0.0.0 @ 32768 ? 
CSR1000v# 


Figure 4. Screenshot of BGP routing table from virtual router prior to traffic filter 


k8s-master1: ~ — ssh root@192.168.31.90 telnet 192.168.31.254 Jr 


CSR1000v#sh ip bgp 

BGP table version is 23, local router ID is 1.1.1.1 

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
x best-external, a additional-path, c RIB-compressed, 

Origin codes: i - IGP, e - EGP, ? - incomplete 

RPKI validation codes: V valid, I invalid, N Not found 


Network Next Hop Metric LocPrf Weight Path 

*> 1.1.1.1/32 0.0.0.0 e 32768 i 
*> 7.7.7.7/32 0.0.0.0 e 32768 ? 
*> 10.0.99.0/24 33.33.33.34 O 65001 ? 
*> 10.1.3.0/24 33.33.33.34 0 65001 ? 
*> 10.1.4.0/24 33.33.33.34 ® 65001 ? 
*> 10.1.5.0/24 33.33.33.34 O 65001 ? 
*> 172.16.0.06/24 33.33.33.34 O 65001 ? 

5 9 8.0.9 (:] 32768 ? 


CSR1000v#show ip route 10.1.6.1 
% Subnet not in table 


Figure 5. Screenshot of BGP routing table from route after traffic filter 


A case analysis for Kubernetes network security of financial service industry in Indonesia ... (Nico Surantha) 


3150 O ISSN: 2302-9285 


The second step is the penetration test of the design from the external route to the application of the 
container network environment. To simulate a reconnaissance attack on the container pods, a port scanning is 
performed with the Nmap tool, initiated from the outside of Kubernetes’ network. The perimeter firewall 
policy is configured according to the zero-trust model. All of the traffic is denied by default and allows only 
web traffic, internet control message protocol (ICMP), and BGP communication traffic for testing purposes. 

Figure 6 signifies that the Nmap scan result is shown in four pods: 1 pod runs MYSQL service with 
the IP address of 10.1.5.1, and 3 pods run the NGINX service with TCP 80 port opened. The port of MYSQL 
pod is opened at TCP 3306. Nonetheless, it results in closed and unidentified status. Since the TCP 3306 port 
communication or the MYSQL service is not allowed by the policy, such a testing result is expected. 
Meanwhile, the MYSQL pod in the actual scenario would not be attained or allowed from any external 
networks because the MYSQL pod should be for the internal route only. Moreover, the MYSQL pod is 
reachable because ICMP communication traffic from the external route to the pod’s network is only allowed 
for testing purposes. 


root@kali:~# nmap -sV -p @-65535 10.1.5.0/24 
Starting Nmap 7.70 ( https://nmap.org ) 

Nmap scan report for 10.1.5.1 

Host is up (@.0012s latency). 

All 65536 scanned ports on 10.1.5.1 are filtered 


Nmap scan report for 10.1.5.2 
Host is up (0.0023s latency). 


Not shown: 65535 filtered ports 
PORT STATE SERVICE VERSION 
80/tcp closed http 


Nmap scan report for 10.1.5.3 
Host is up (0.0026s latency). 

Not shown: 65535 filtered ports 
PORT STATE SERVICE VERSION 
8@/tcp open http nginx 1.9.11 


Nmap scan report for 10.1.5.4 
Host is up (@.0026s latency). 

Not shown: 65535 filtered ports 
PORT STATE SERVICE VERSION 
8@/tcp open http nginx 1.9.11 


Nmap scan report for 10.1.5.5 
Host is up (@.0625s latency). 

Not shown: 65535 filtered ports 
PORT STATE SERVICE VERSION 
8@/tcp open http nginx 1.9.11 


Service detection performed. Please report any incorrect results at https://nmap.org/submit/ 
Nmap done: 256 IP addresses (5 hosts up) scanned in 337.77 seconds 
root@kali:~# 


Figure 6. Screenshot of port scanning from external to container network 


The third step is to evaluate the conformability of the proposed design to the other four technical 
requirements. Table 5 summarizes all of the evaluation results, which provide a better perspective of this 
particular work. To evaluate the internal network security, the label assignment to the pods verifies the 
container’s pod-to-pod communication. Although the two pods are in the same network segment and 
namespace, the ICMP test traffic can be inspected and denied due to undefined communication in the 
distributed internal firewall’s policy. The default firewall policy is configured to deny all traffic in the zero 
trust policy. In other words, communication would not be permitted without permission from the firewall 
policy. The traffic inspection was implemented by Kubernetes label that identifies the pods of incoming 
traffic to logical switch in the NSX networking. The policy only allows a legitimate pod-to-pod 
communication traffic so that the built-in internal firewall is identified and inspected. By default, pods could 
not communicate with another pod since they are not allowed and denied. Then, the logs of the inspected 
traffic are forwarded to an external Syslog server, and the ejected logs by the firewall are displayed. 
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5. CONCLUSION 

A simulation of the design was built to evaluate the secure network design and validate network 
security following the zero trust model. To ensure proper implementation of firewall policy at the perimeter, 
penetration testing of network security, such as external network scanning and route hijacking, was 
conducted. The results show that the network is not penetrated. To examine the internal security, the internal 
traffic was identified using Kubernetes’ labels, which allow the internal security to flow in the environment. 
The default has restricted non-legitimate or unidentified traffic. All of the traffic is examined by the firewall 
so that the traffics are logged to an external Syslog server to store the connectivity information. Then, the 
traffics could be repurposed for troubleshooting, forensics, and analysis. For future work, several aspects 
could be considered to improve the design. First, the role-based access controls could be implemented to 
provide strict control and the least privilege for authorized users. Second, identity management systems could 
be applied by unauthorized users and the next possible security-related users and would prevent unauthorized 
access. 
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